Hits and misses: Kirby on the selection task.

Kirby (1994) has helpfully applied the terminology of signal detection theory to Watson's selection task. We agree with him that it is necessary to bring out the decision theoretic aspects of the selection task (Evans, Over & Manktelow, 1993), and Kirby makes an important contribution by doing this so fully for an abstract version of the task. However, there are problems with both the design of Kirby's experiments and the interpretation which he places upon them. Kirby discusses his abstract selection tasks in a rather abstract way, but his points can be made with more intuitive examples. More importantly, these points can be related to fundamental issues in the philosophy of science, particularly those stressed by Bayesian thinkers. Using a standard intuitive example, we can bring out the general interest of Kirby's work as well as some of its difficulties. (See Howson & Urbach, 1989, for a good discussion in the philosophy of science of essentially the example we shall use. Wetherick, 1993, notes the relevance of this example to the selection task.) Consider the standard example in the philosophy of science, expressed in a conditional form: if it is a raven then it is black. Suppose we were given the task of finding out whether this statement, taken as one about all things, can be falsified. To do this, we would be much better advised to go out looking for ravens than for non-black things. It is true that we would have a falsifying case if we found a non-black thing which was a raven. But the set of ravens is small relative to that of non-ravens, and thus any non-black thing we find is almost bound to be a non-raven in any case. It is better to search through the relatively circumscribed set of ravens. Following Kirby's lead, we can neatly describe the possible results of this search in the terminology of signal detection theory. For the task we were given, finding a raven which is non-black is a hit, but finding a raven which is